Introduction: Multimodal emotion recognition due to receiving information from different sensory resources (modalities) from a video has a lot of challenges and has attracted many researchers as a new method of human computer interaction. The purpose of this paper was to automatically recognize emotion from emotional speech and facial expression based on the neural mechanisms of the brain. Therefore, based on studies on brain-inspired models, a general framework for bimodal emotion recognition inspired by the functionality of the auditory and visual cortics and brain limbic system is presented. Methods: The hybrid and hierarchical proposed model consisted of two learning phases. The first step: the deep learning models for the representation of visual and auditory features, and the second step: a Mixture of Brain emotional Learning (MoBEL) model, obtained from the previous stage, for fusion of audio-visual information. For visual feature representation, 3D-convolutional neural network (3D-CNN) was used to learn the spatial relationship between pixels and the temporal relationship between the video frames. Also, for audio feature representation, the speech signal was first converted to the log Mel-spectrogram image and then fed to the CNN. Finally, the information obtained from the two above streams was given to the MoBEL neural network model to improve the efficiency of the emotional recognition system by considering the correlation between visual and auditory and fusion of information at the feature level. Results: The accuracy rate of emotion recognition in video in the eNterface’ 05 database using the proposed method was on average of 82%. Conclusion: The experimental results in the database show that the performance of the proposed method is better than the hand-crafted feature extraction methods and other fusion models in the emotion recognition.

Feature selection is one of the best optimization problems in human recognition, which reduces the number of features, removes noise and redundant data in images, and results in high rate of recognition. This step affects on the performance of a human recognition system. This paper presents a Multimodal biometric verification system based on two features of palm and ear which has emerged as one of the most extensively studied research topics that spans multiple disciplines such as pattern recognition, signal processing and computer vision. Also, we present a novel Feature selection algorithm based on Particle Swarm Optimization (PSO). PSO is a computational paradigm based on the idea of collaborative behavior inspired by the social behavior of bird flocking or fish schooling. In this method, we used from two Feature selection techniques: the Discrete Cosine Transforms (DCT) and the Discrete Wavelet Transform (DWT). The identification process can be divided into the following phases: capturing the image, pre-processing, extracting and normalizing the palm and ear images, feature extraction, matching and fusion, and finally, a decision based on PSO and GA classifiers. The system was tested on a database of 60 people (240 palm and 180 ear images). Experimental results show that the PSO-based feature selection algorithm was found to generate excellent recognition results with the minimal set of selected features.

Introduction: Nowadays, the use of artificial intelligence and machine learning has impacted all fields of study. Utilizing these methods for identifying individuals' emotions through integrating audio, text, and image data has shown higher accuracy than conventional methods, presenting various applications for psychologists and human-machine interaction. Identifying human emotions and individuals' reactions is crucial in psychology and psychotherapy. emotional identification has traditionally been conducted individually and by analyzing facial expressions, speech patterns, or handwritten responses to stimuli and events. However, depending on the subject's conditions or the analyst's circumstances, this approach may lack the required accuracy. This paper aimed to achieve high-precision emotional recognition from audio, text, and image data using artificial intelligence and machine learning methods. Methods: This research employs a correlation-based approach between emotions and input data, utilizing machine learning methods and regression analysis to predict a criterion variable based on multiple predictor variables (the emotional category as the criterion variable and the features, audio, image, and text variables as predictors). The statistical population of this study is the IEMOCAP dataset, and the data type of this research is a mixed quantitative-qualitative approach. Results: The results indicated that combining audio, image, and text data for multi-modal emotional recognition significantly outperformed the recognition of emotions from each data alone, exhibiting a precision of 82.9% in the baseline dataset. Conclusions: The results demonstrate a considerably acceptable precision in identifying human emotions through audio integration, text, and image data compared to individual data when using machine learning and artificial intelligence methods.

emotion Speech recognition (ESR) is recognizing the formation and change of speaker’ s emotional state from his/her speech signal. The main purpose of this field is to produce a convenient system that is able to effortlessly communicate and interact with humans. The reliability of the current speech emotion recognition systems is far from being achieved. However, this is a challenging task due to the gap between acoustic features and human emotions, which relies strongly on the discriminative acoustic features extracted for a given recognition task. Deep learning techniques have been recently proposed as an alternative to traditional techniques in ESR. In this paper, an overview of Deep Learning techniques that could be used in emotional Speech recognition is presented. Different extracted features like MFCC as well as feature classifications methods including HMM, GMM, LTSTM and ANN have been discussed. In addition, the review covers databases used, emotions extracted, and contributions made toward ESR.

emotion has an important role in naturalness of man-machine communication and many researchers investigate computerized emotion recognition from speech in the recent decades. In this paper, the effect of formant-related features on improving the performance of emotion detection systems is experimented. To do this, various forms and combinations of the first three formants are concatenated to a popular feature vector and Gaussian mixture models are used as classifiers. Experimental results show average recognition rate of 69% in four emotional states and noticeable performance improvement by adding only one formant-related parameter to feature vector. The architecture of hybrid emotion recognition/spotting is also proposed based on the developed models.